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This  paper  describes  a  precision-guided  surgical  navigation  system  for  minimally  invasive  surgery.  The 
system  combines  a  laser  guidance  technique  with  a  three-dimensional  (3D)  autostereoscopic  image  over¬ 
lay  technique.  Images  of  surgical  anatomic  structures  superimposed  onto  the  patient  are  created  by 
employing  an  animated  imaging  method  called  integral  videography  (IV),  which  can  display  geomet¬ 
rically  accurate  3D  autostereoscopic  images  and  reproduce  motion  parallax  without  the  need  for  special 
viewing  or  tracking  devices.  To  improve  the  placement  accuracy  of  surgical  instruments,  we  integrated 
an  image  overlay  system  with  a  laser  guidance  system  for  alignment  of  the  surgical  instrument  and  better 
visualization  of  patient’s  internal  structure.  We  fabricated  a  laser  guidance  device  and  mounted  it  on  an 
IV  image  overlay  device.  Experimental  evaluations  showed  that  the  system  could  guide  a  linear  surgical 
instrument  toward  a  target  with  an  average  error  of  2.48  mm  and  standard  deviation  of  1.76  mm.  Further 
improvement  to  the  design  of  the  laser  guidance  device  and  the  patient-image  registration  procedure  of 
the  IV  image  overlay  will  make  this  system  practical;  its  use  would  increase  surgical  accuracy  and  reduce 
invasiveness. 

©  2009  Elsevier  Ltd.  All  rights  reserved. 


1.  Introduction 

Image-guided  surgery  is  a  means  of  navigation  that  guides  the 
surgeon  by  indicating  the  location  of  a  tracking  device  through 
a  set  of  cross-sectional  images,  for  example,  X-ray  computed 
tomography  (CT)  and/or  magnetic  resonance  (MR)  images,  or  three- 
dimensional  (3D)  anatomical  models  reconstructed  from  such 
images.  By  localizing  the  target  and  the  critical  lesion  that  should 
be  avoided,  surgical  navigation  helps  to  achieve  effective  and  safe 
surgery  while  minimizing  the  invasiveness  of  the  surgery  [1  ]. 

In  conventional  surgical  operations,  the  display  of  the  surgical 
navigation  system  is  often  placed  in  a  nonsterile  field  away  from  the 
surgeon.  The  consequent  hand-eye  coordination  problem  forces 
the  surgeon  to  take  extra  steps  to  match  the  guidance  information 
on  the  display  with  the  actual  anatomy  of  the  patient  [2],  which  can 
be  time  consuming,  especially  when  the  target  is  small  and  deeply 
located.  The  problem  is  a  possible  interruption  in  the  surgical  flow 
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[3].  To  overcome  this  problem,  several  research  groups  have  devel¬ 
oped  techniques  that  merge  images  into  real-world  views  in  a  more 
natural  and  unconstrained  manner.  These  techniques  involve  an 
image  overlay  that  visually  merges  a  computer-generated  anatom¬ 
ical  model,  reconstructed  from  medical  images,  with  the  patient’s 
body  [4-6].  Another  example  is  the  use  of  augmented  reality  visu¬ 
alization  with  microscopes  or  endoscopes  in  image-guided  surgery 
[7,8]. 

Most  studies  on  augmented  reality  in  surgical  environments 
have  focused  on  the  use  of  virtual  reality  operative  microscopes, 
head-mounted  displays  (HMD),  and  semi-transparent  mirrors. 
These  image-overlay  studies  focused  on  how  to  overlay  pre-/intra- 
operative  images  on  the  patient.  Few  came  close  to  resolving  the 
problem  of  intuitively  guiding  surgical  instruments.  Certain  sur¬ 
gical  treatments  such  as  those  of  orthopedic  surgery,  requires 
high  position  and  orientation  accuracy  when  the  surgical  tools  are 
inserted  into  the  patient.  For  example,  during  knee  joint  surgery,  the 
surgeon  needs  to  identify  both  the  surgical  area  and  the  insertion 
path  for  tunneling  or  drilling. 

Researchers  in  the  1990s  introduced  the  use  of  lasers  for  surgi¬ 
cal  instrument  guidance.  Miaux  et  al.  developed  a  laser  guidance 
system  for  CT-guided  procedures  [9],  and  Pereles  et  al.  introduced 
a  laser  guidance  system  for  percutaneous  musculoskeletal  proce- 
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dures  [10,11].  These  systems  enable  needle  placement  to  be  more 
accurate,  thus  saving  time  compared  to  a  freehand  approach  by 
minimizing  the  need  for  repositioning.  The  desired  puncture  angle 
is  defined  by  the  laser  beam.  The  needle  is  aligned  with  the  beam, 
and  throughout  the  procedure,  the  laser  dot  should  shine  at  the  hub 
of  the  needle  to  indicate  correct  alignment. 

Sasama  et  al.  developed  a  laser  guidance  system  that  was  mono- 
lithically  integrated  with  an  optical  tracker  [  12,13  ].  The  laser  beams 
are  projected  onto  the  surgical  field  so  that  the  surgeon  can  obtain 
guidance  information  directly.  Glossop  et  al.  used  scanned  infrared 
and  visible  lasers  to  project  computer  generated  information  onto 
the  patient  by  moving  a  laser  beam  in  arbitrary  directions  by  means 
of  controlled  mirrors  [14].  These  systems  still  cannot  provide  intu¬ 
itive  visualization  of  complex  anatomical  structures  of  inside  the 
patient’s  body.  Moreover,  their  devices  are  located  at  a  distance 
from  the  surgical  area  and  sometimes  cause  interruptions  during 
the  operation. 

To  improve  the  safety  and  accuracy  of  surgical  treatments,  espe¬ 
cially  those  using  linear  surgical  tools  for  puncturing  and  drilling, 
we  developed  a  3D  autostereoscopic  image  overlay  system  inte¬ 
grated  with  a  laser  guidance  device.  Two  lasers  are  reflected  by 
mirrors,  and  their  projection  directions  are  controlled  by  motors; 
this  arrangement  gives  more  freedom  to  the  laser  guidance  beams. 
The  combination  of  image  overlay  and  laser  guidance  enables  both 
visualization  of  the  patient  and  guidance  for  the  surgical  instru¬ 
ment. 

2.  Materials  and  methods 

2.1.  Laser  guidance  and  procedure 

The  laser  guidance  system  includes  two  lasers,  which  project 
individual  laser  planes  that  intersected  as  a  line  in  space  (Fig.  1). 
The  plane  of  laser  light  is  produced  by  a  high-quality  compact  line- 
laser  module.  The  laser  shooter  modules  are  fixed  to  motors  on  a 
stand.  The  orientation  of  the  projected  laser  plane  can  be  adjusted 
by  rotation  of  the  motor.  Furthermore,  we  use  mirrors  to  reflect  the 
beams  to  the  target  area.  The  angles  of  the  mirrors  are  also  con¬ 
trolled  by  motors  so  that  the  beams  can  be  adjusted  to  the  required 
position  and  direction  onto  the  target.  Two  laser-beam  planes  are 
projected  onto  the  surgical  area,  and  the  projection  directions  are 
controlled  by  the  navigation  system  on  a  PC.  The  rotation  angles  of 
the  motors  are  limited  so  that  the  lasers  do  not  project  outside  of 
the  required  area. 

The  intersection  line  generated  by  the  laser  planes  is  used  to 
guide  the  insertion  path  of  a  linear  surgical  instrument.  Fig.  2  shows 
the  procedure  for  aligning  the  surgical  instrument. 


(1 )  Two  laser  planes  cross  on  the  surgical  area  (Fig.  2a).  The  point  of 
intersection  of  these  planes  on  the  target  area  is  determined  by 
using  the  calibration  and  patient  registration  method  described 
in  Sections  2.4  and  2.5. 

(2)  Move  the  tip  of  the  surgical  instrument  to  the  crossing  point 
on  the  target  area  (Fig.  2b).  The  position  of  the  crossing  point 
should  be  calculated  according  to  the  3D  shape  of  the  anatom¬ 
ical  structure. 

(3)  Rotate  the  surgical  instrument  around  the  crossing  point  and 
align  it  to  the  insertion  path  (Fig.  2c).  Adjust  the  orientation  of 
the  surgical  instrument  till  the  laser  beam  shines  on  top  of  it. 
The  proper  orientation  can  be  determined  by  putting  a  coaxial 
point  on  the  tail  of  the  surgical  instrument  or  by  putting  parallel 
straight  lines  on  the  cylindrical  surface  of  it  (Fig.  2d). 

(4)  Stabilize  the  surgical  instrument  after  obtaining  the  surgical 
path  by  the  aligning  it  with  the  insertion  point  and  orienta¬ 
tion.  Confirm  the  correct  position  by  repeatedly  scanning  with 
the  surgical  instrument,  if  necessary. 

2.2.  IV  autostereoscopic  image  overlay 

The  autostereoscopic  image  overlay  technique  can  be  integrated 
into  a  surgical  navigation  system.  This  technique  superimposes  an 
actual  3D  image  onto  the  patient  from  the  perspective  of  the  sur¬ 
geon  by  using  a  half-silvered  mirror.  The  autostereoscopic  image  is 
generated  by  using  a  version  of  integral  videography  (IV)  [15,16], 
which  is  an  animated  extension  of  the  integral  photography  [17]. 

The  IV  display  we  developed  consists  of  a  high-resolution  2-D 
LCD  with  a  microconvex  lens  array.  The  computer-generated  images 
of  the  anatomical  object  are  rendered  behind  the  lens  array.  The 
source  image  shown  on  the  display  is  generated  from  the  3D  data 
by  high-speed  IV  rendering.  Any  kind  of  3D  data  source,  such  as  MRI 
and  CT  images,  can  be  processed.  Each  point  shown  in  the  3D  space 
is  reconstructed  at  the  same  position  as  the  actual  object  through 
the  convergence  of  rays  from  the  pixels  of  the  element  images  on 
the  computer  display  after  they  pass  through  the  lenses  of  array.  The 
surgeon  can  see  any  point  on  the  display  from  various  directions,  as 
if  it  were  in  3D  space.  Each  point  appears  as  a  different  light  source, 
and  a  3D  object  is  thus  constructed  as  an  assembly  of  reconstructed 
light  sources  [15].  Since  the  element  images  are  in  the  focal  plane 
of  the  lens,  only  one  pixel  can  be  observed  from  each  lens.  The 
observed  lateral  resolution  of  the  IV  image  is  proportional  to  the 
number  of  lenses.  On  the  other  hand,  since  viewpoints  equal  to  the 
number  of  the  pixels  are  covered  by  one  lens,  the  vertical  resolution 
is  determined  by  the  pixel  number.  The  quality  of  the  IV  image 
depends  primarily  on  the  pixel  density  of  the  2-D  display  and  the 
lens  pitch  of  the  array.  To  improve  image  quality,  both  the  pixel  pitch 
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Fig.  1.  (a)  Laser  guidance  by  using  two  intersecting  laser  beam  planes,  (b)  Configuration  of  laser  guidance  device. 
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Fig.  2.  Procedure  of  surgical  instrument  alignment:  (a)  target  in  surgical  area;  (b)  move  the  tip  of  surgical  instrument  to  the  insertion  point;  (c)  rotate  surgical  instrument 
and  adjust  its  orientation;  (d)  align  surgical  instrument  to  the  target  orientation  by  using  coaxial  point  or  parallel  projection  lines. 


of  the  element  images  and  the  lens  pitch  must  be  made  smaller. 
Furthermore,  the  error  between  the  lens  pitch  and  the  width  of  the 
element  image  causes  the  deformation  that  may  significantly  affect 
the  perceived  depth  perception  of  deep  locations.  Pixels  adjustment 
and  image  correction  o  must  be  performed  before  the  IV  display  is 
used. 

The  main  elements  of  the  overlay  system  include  the  intuitive 
3D  image  on  the  IV  display,  a  half-silvered  mirror,  a  reflected  spatial 
3D  image,  and  a  supporting  stage  [15].  A  surgeon  observes  the  oper¬ 
ating  field  directly  by  eye  during  an  operation.  To  ensure  smooth 
operation,  the  surgeon  must  be  able  to  identify  the  target  object, 
avoid  specific  areas  (critical  areas),  and  respond  to  intra-operative 
information  when  it  is  presented. 

2.3.  System  of  IV  image  overlay  with  laser  guidance 

The  surgical  navigation  system  consists  of  the  IV  display,  the 
laser  guidance  device,  3D  data  source  collection  equipment,  an 
optical  position  tracking  device  (POLARIS,  Northern  Digital  Inc., 
Waterloo,  Ontario,  Canada)  and  a  personal  computer,  as  shown  in 
Fig.  3.  The  laser  guidance  device  is  mounted  at  the  bottom  of  the 
IV  image-overlay  device.  The  distance  between  the  laser  shooter 
and  the  patient  is  about  30  cm.  The  projection  direction  of  the  laser 
plane  is  adjusted  by  a  motor  rotating  the  mirror  axis.  The  optical 
position  tracking  system  is  used  for  both  calibration  of  the  laser 
guidance  device  and  registration  of  the  IV-image  to  patient.  To 
project  the  laser  beams  at  the  proper  position  and  orientation,  we 


calibrate  the  position  and  orientation  of  the  intersection  line  and 
calculate  the  relationship  between  the  reflection  direction  of  the 
mirror  and  the  rotation  of  the  motor.  For  this,  a  needle  or  surgi¬ 
cal  instrument  is  inserted  superficially  into  the  point  of  entry  of 
the  patient  and  aligned  to  the  intersection  line  of  the  laser  beam 
planes. 

Fig.  4  shows  a  flowchart  illustrating  the  IV  image  overlay  with 
the  laser  guidance.  Performing  an  image-guided  surgery  involves 
two  important  steps:  (1)  Initialize  the  IV  image  overlay  device  and 
register  the  image  to  the  patient.  Using  this  image  overlay,  the  sur¬ 
geon  can  see  through  the  patient’s  body  intuitively.  (2)  Initialize  the 
laser  guidance  device  and  control  the  laser  beams  to  correspond¬ 
ing  directions.  With  this  laser  guidance,  the  surgeon  can  visualize 
the  insertion  position  and  path  of  the  surgical  instrument  directly. 
If  intra-operative  information  of  patient  and  surgical  instrument  is 
updated,  we  will  perform  the  IV  image  registration  and  correspond¬ 
ing  laser  guidance  again. 

2.4.  Calibration  of  laser  guidance  device 

The  projected  directions  and  reflected  angles  of  the  laser  beam 
planes  are  calculated  on  the  basis  of  the  required  position  and  ori¬ 
entation  of  the  surgical  instrument.  The  laser  shooter  is  in  a  sheath 
firmly  connected  to  one  of  the  motors.  The  procedure  of  adjusting 
the  axis  of  the  laser  shooter  and  the  direction  of  the  projected  line 
laser  is  somewhat  complicated.  The  calibration  process  consists  of 
the  following  steps. 


Fig.  3.  System  configuration  of  IV  image  overlay  with  laser  guidance. 
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Fig.  4.  Procedure  of  IV  image  overlay  with  laser  guidance. 


(1)  Adjust  the  direction  of  the  laser  shooter  to  make  its  axis 
parallel  to  the  axis  of  the  motor.  For  this,  we  insert  two 
mounts  into  the  sheath  and  adjust  the  position  of  the  laser 
shooter.  This  alignment  is  laborious  and  does  not  yield  pre¬ 
cise  alignments.  A  software  based  calibration  method  is  being 
developed  to  correct  any  misalignment  of  the  axis  of  the  laser 
shooter. 

(2)  Calibrate  the  movements  of  the  motors  with  angles  of  the 
mirrors.  Calculate  the  relationship  between  the  reflected  laser 
plane  and  the  angle  of  the  mirrors  so  that  the  rotation  angles 
can  be  thus  controlled  by  the  stepping  motors. 

(3)  Calibrate  the  position  and  orientation  of  the  intersection  line  of 
the  two  laser  beam  planes.  A  set  of  test  laser  beams  are  projected 
onto  a  plate  in  different  directions.  The  3D  spatial  positions  of 
the  projected  laser  beams  are  measured  with  the  optical  track¬ 
ing  device.  Accordingly,  the  relationship  between  the  projected 
laser  beam  plane  and  the  rotation  angles  of  the  mirror  and  the 
motors  can  be  determined. 


The  laser  guidance  devices  are  fixed  in  the  IV  overlay  device, 
which  can  be  also  tracked  by  the  same  optical  tracking  device.  Thus 
we  can  calculate  the  transformation  matrix  between  the  original 
laser  plane  and  the  optical  tracking  device. 

The  rotation  angles  of  the  mirrors  and  the  laser  shooters  are 
determined  as  follows.  The  coordinates  of  the  system  are  shown  in 
Fig.  5.  First,  we  consider  the  right-side  laser  module.  We  set  rota¬ 
tions  that  represent  a  counterclockwise  rotation  of  angle  oq  around 
the  axis  of  motor  R-t  (the  motor  axis  is  parallel  to  the  x-axis). 
The  normal  direction  of  the  projected  laser  beam  plane  becomes 
=  Rx{a^  )n0,  where 


Rx(0)  = 


1  0  0 
0  cos 0  -sin# 
0  sin#  cos  0 


Given  a  unit  vector  u  =  (ux,uy,  uz\  the  matrix  for  a  counterclock¬ 
wise  rotation  through  an  angle  of  0  about  an  axis  the  direction  of  u 


is  R(u,  0)  =  P  +  (/  -  P)  cos  0  +  Q  sin  0;  where 


’  "I 

UXUy 

uxuz~ 

0 

-uz 

Uy  " 

p  = 

UXUy 

uj 

UyUZ 

=  uuT,  Q  = 

Uz 

0 

-ux 

uxuz 

UyUZ 

u2z  _ 

Ux 

0 

and  /  is  a  3  x  3  identity  matrix. 

Let  the  original  normal  vector  of  the  mirror  be  nm0,  and  the  direc¬ 
tion  of  the  rotation  axis  be  hr0.  We  first  rotate  the  axis  of  Motor  R-2 
(on  the  mirror  plane)  clockwise  through  an  angle  0 1  around  the  z- 
axis.  The  direction  of  the  axis  of  Motor  R-2  is  given  by  nr  =  Pz(0i  )nro, 
where 


RztO)  = 


cos 0  -sin#  0 
sin#  cos  0  0 

0  0  1 


After  rotating  the  mirror  through  an  angle  of  /3i  around  the 
axis  of  Motor  R-2 ,  the  normal  vector  of  the  mirror  changes  to 
=R(nr,  )rim o- 

When  the  laser  beam  is  reflected  by  the  mirror  rotated  by  ^ , 
the  normal  vector  of  the  laser  beam  plane  will  change  to  n\  =ri\  - 
2  nm{nm  •  Hi  )•  The  normal  direction  of  the  left-side  laser  beam  plane 
h'2  can  be  also  derived  in  the  same  way. 

After  initializing  the  laser  guidance  device  and  fixing  the  device 
in  the  operation  space,  we  can  obtain  the  position  of  the  rotation 
center  (the  intersection  point  of  the  two  motors  axes)  0\  and  02 
of  each  laser  beam.  The  laser  beam  planes  can  be  determined  from 
the  normal  direction  of  the  plane  and  the  rotation  center  as  (Xi  - 
0! )  •  n\  =  0,  and  (X2  -  02)  •  h'2  =  0,  where  X!  and  X2  are  points  on 
the  two  laser  beam  planes.  The  vector  of  the  intersection  line  of  the 
two  laser  beam  planes  is  therefore  given  by  h  =  h\  x  h'2. 

From  the  required  insertion  position  and  orientation  of  the  laser 
line  determined  by  the  patient-image  registration  described  in  next 
section,  we  can  calculate  the  rotation  angle  of  the  laser  shooter  and 
the  mirror  from  the  above  derivation. 


Fig.  5.  Calculate  rotation  angles  of  motors  according  to  the  insertion  position  and 
orientation. 
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2.5.  Combining  of  laser  guidance  and  IV  image  overlay 

The  registration  of  the  surgical  navigation  system  includes  the 
IV  image  overlay  and  laser  guidance  to  the  anatomical  objects. 
The  relationships  between  coordinates  are  shown  in  Fig.  6.  Opti¬ 
cal  markers  attached  to  the  overlay  device  are  used  for  both 
patient-to-image  registration  and  laser-to-patient  registration.  To 
combine  the  IV  image  with  the  line  formed  by  the  intersect¬ 
ing  laser  planes,  the  calibration  and  registration  process  is  as 
follows. 

To  calibrate  the  IV  image,  we  relate  the  position  of  the  IV  display 
and  the  coordinates  of  the  optical  markers,  M. 

(1)  The  calibration  of  the  IV  images  on  the  display,  D,  can  be  per¬ 
formed  by  projecting  a  reference  3D  model  with  calibration 
points  on  it  and  reflecting  the  IV  images  into  the  3D  space  via 
the  half-slivered  mirror  [18].  We  calculate  the  positions  of  the 
reflected  IV  images,  producing  transformation  ^ T . 

(2)  Calibrate  the  rigid  body  of  the  optical  position  markers  attached 
to  the  IV  overlay  device,  relative  to  the  optical  tracking  system, 
0,  and  produce  transformation  ^  T .  For  this,  we  can  get  the  rela¬ 
tionship  between  the  IV  display  and  the  optical  tracking  system 

Dt _ M'rD  rr 

0 1 2 3 4 5 6  -  0  1  M1  ' 

Regarding  the  laser-plane  intersection  line,  since  the  laser 
guidance  device  is  fixed  to  the  IV  image  overlay  device,  the 
same  optical  markers  attached  to  the  overlay  device  give  the 
following  transformations: 

(3)  Calculate  the  laser  beam  planes  and  corresponding  laser  line,  L, 
producing  transformation  ^ T . 

(4)  Using  ^T,  get  the  relationship  between  the  laser  line  and  the 
optical  tracking  system:  f  T  =  ^Tj^T. 

(5)  Using  the  calibration  described  in  Section  2.4,  get  the  relation¬ 
ship  between  the  rotation  angle  of  the  motors  and  the  laser 
line. 

For  the  patient-to-image  and  patient-to-laser  registrations, 
we  use  a  set  of  distinct  features  (geometrical  landmarks  or  nat¬ 
ural  anatomical  landmarks)  that  can  be  accurately  identified  in 
both  the  scanned  image  (CT,  MRI,  etc.)  and  the  patient’s  physi¬ 
cal  space,  P,  to  relate  the  coordinates  of  the  spaces.  Accordingly, 
the  geometric  transformations  that  associate  these  three  spaces 
(patient,  IV  image,  and  laser)  can  be  derived. 

(6)  The  transformation  of  the  IV  image  for  patient-to-image  over¬ 
lay  is  given  by  £ T  =  ^T^T^T,  and  the  laser  line  for  guidance  is 

therefore  given  by  LpT  =  LMT^T°T  =  LMT(°Tf  '°T. 


After  finishing  patient-image  registration,  we  drive  the  motors 
to  rotate  the  axes  of  the  laser  shooter  and  the  mirror  to  calcu¬ 
lated  angles.  The  insertion  path  for  the  surgical  instrument  is  along 
the  laser-plane  intersection  line.  The  target  objects  are  represented 
in  the  form  of  IVs,  which  are  displayed  in  real-time  on  the  IV 
display. 

3.  Experiments  and  results 

3.1.  Laser  guidance  devices 

We  fabricated  a  laser  guidance  device  (Fig.  7)  composed  of 
two  laser  shooters  (Edmund,  VLM™  3)  and  four  stepping  motors 
(PK513PA,  ORIENTAL  MOTOR  Co.  LTD,  Japan),  and  we  mounted  it 
on  the  bottom  of  the  IV  image  overlay  device.  This  sort  of  mount 
does  not  influence  the  implementation  of  controlling  the  surgical 
instrument  during  the  operation.  The  range  of  projection  angle  of 
the  laser  beam  plane  is  30°.  The  orientation  of  each  laser  beam 
plane  is  controlled  by  two  motors.  One  motor  rotates  the  axis  of 
the  projected  laser  beam.  The  other  motor  adjusts  the  reflection 
angle  of  the  mirror  so  that  the  laser  plane  projects  to  the  surgical 
field  (Fig.  7a).  The  projected  laser  beams  are  illustrated  in  Fig.  7b.  A 
set  of  optical  tracking  markers  is  attached  in  the  base  of  the  laser 
guidance  device. 

3.2.  Accuracy  of  laser  guidance 

We  carried  out  an  experiment  to  evaluate  the  accuracy  of  the 
laser  guidance.  We  calibrated  the  laser  guidance  device  with  a  set  of 
test  patterns.  The  experiment  was  performed  based  on  the  calibra¬ 
tion  and  correction  results  for  the  relationship  between  the  motors 
and  laser  projection  planes.  We  used  the  POLARIS  optical  tracking 
device  to  measure  the  projected  crossing  point  and  the  intersection 
line  of  the  two  laser  beam  planes. 

The  procedure  is  as  follows.  We  attached  an  optical  tracking  sen¬ 
sor  to  the  surgical  tool.  The  position  and  situation  of  the  surgical  tool 
could  be  thus  obtained  by  the  tracking  system.  We  chose  a  set  of 
text  patterns  that  included  the  insertion  positions  and  orientations. 
For  each  evaluation,  we  moved  the  tool  to  the  required  position 
and  orientation  and  adjusted  according  to  the  tracking  results  of 
the  POLARIS  system.  We  calculated  the  rotation  angles  of  the  laser 
shooters  and  the  mirrors,  and  controlled  the  motors  and  rotated  the 
two  laser  beam  planes.  After  finishing  the  laser  beam  rotation,  we 
moved  the  same  tool  to  the  crossing  point  and  adjusted  the  pos¬ 
ture  of  the  tool  according  to  laser  guidance  method  described  in 
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Fig.  7.  Laser  guidance  devices:  (a)  motors  and  mirrors  for  adjusting  the  orientation  of  the  laser  beam  planes;  (b)  laser  planes  intersect  to  form  a  line. 


Fig.  8.  An  optical  tracking  sensor  attached  to  surgical  tool  for  tracking  insertion 
position  and  orientation  of  the  tool. 


Section  2.1 .  We  measured  the  position  of  the  crossing  point  and  the 
orientation  of  the  tool  (Fig.  8). 

The  mean  value  of  the  position  error  of  the  insertion  point  for 
25  tests  was  2.48  mm,  and  the  standard  deviation  was  1.76  mm. 
The  mean  value  of  the  orientation  error  of  the  intersection  line  was 
2.96°,  and  the  standard  deviation  was  2.12°. 

We  conducted  another  set  of  experiments  to  assess  the  fea¬ 
sibility  of  aligning  surgical  instrument  with  an  insertion  path 
determined  by  the  laser  planes  (Fig.  9).  After  deciding  the  inser¬ 
tion  point  and  orientation,  the  laser  beam  planes  will  be  rotated  to 


Table  1 

Main  specifications  of  IV  overlay  device. 


Setup 

Specifications 

Characteristic 

IV  display 

Size  of  display 

Number  of  pixels 

Pixel  pitch 

Lens  pitch 

Lens  focal  length 

Lens  refractive  index 

129.0  mm  x  96.7  mm 

1024  x  768  pixels 

0.127  mm 

1.001  mm  (H),  0.876  mm  (V); 
(hexagon) 

3.0  mm 

Nd  =  1.49 

Overlay  device 

Viewing  angle 
Projection  distance 
Weight 

Size 

About  ±10° 

300  mm  (230  mm-370  mm) 

2080g 

175  mm  (L)  x  370  mm 
(W)  x  150  mm  (H) 

Laser  guidance  device 

Weight 

Size 

580g 

170  mm  (L)  x  110  mm 
(W)  x  50  mm  (H) 

Total  device 

Weight 

2660g 

the  calculated  directions  to  intersect  and  form  the  guidance  laser 
line.  This  line  makes  it  easy  to  guide  a  surgical  instrument. 

3.3.  IV  image  overlay 

We  manufactured  an  IV  image  overlay  device  with  a  micro- 
convex  lens  array,  a  half-silvered  mirror,  and  a  supporting  stage 
(Fig.  10a).  The  main  specifications  of  the  IV  display  are  listed  in 
Table  1.  The  rendering  time  of  the  IV  image  heavily  depended  on 
the  complexity  of  the  rendered  object.  We  used  a  Graphic  Pro¬ 
cessing  Unit  (GPU)-accelerated  IV  rendering  method  to  enable 


Fig.  9.  Alignment  of  surgical  instrument  to  path  determined  by  laser-plane  intersection:  (a)  move  the  tip  of  the  surgical  instrument  to  the  crossing  point  of  the  laser  planes; 
(b)  adjust  the  orientation  of  the  surgical  instrument;  (c)  identify  coaxial  point  and  parallel  projection  lines  on  the  surgical  instrument  and  stabilize  the  surgical  instrument. 
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Fig.  10.  (a)  IV  image  overlay  device;  (b)  image-overlay  result  for  the  knee  joints. 


Fig.  11.  Example  of  laser  guidance  with  autostereoscopic  image  overlay:  (a)  IV  image 
(c)  image-patient  registration  results  and  surgical  path  guidance  of  laser  beams. 

real-time  IV  rendering.  Despite  the  complexity  of  the  data,  the 
method  enabled  an  update  speed  of  14  frames  per  second  for  a 
data  of  256  x  256  x  256  voxels  worth  of  data  (on  an  Intel  Pentium 
4, 3.2  GHz  CPU).  The  IV  image  could  be  calculated  and  displayed  in 
almost  real-time  with  our  optimized  algorithm. 


overlay  device  and  patient  with  IV  image  overlay;  (b)  alignment  of  surgical  instrument; 


The  IV  image  could  be  superimposed  onto  the  patient  with  fidu¬ 
cial  marker  based  registration.  Once  all  objects  to  be  displayed  are 
transformed  into  the  same  coordinate  space,  they  appear  to  the  sur¬ 
geon  exactly  as  if  they  were  in  their  actual  spatial  locations.  One  of 
the  overlaid  IV  images  is  shown  in  Fig.  10b. 


Fig.  12.  (a)  Observe  the  surgical  instrument  aligned  with  the  intersected  laser  line  from  side,  (b)  Observe  from  the  viewing  windows  of  the  IV  image  overlay  device. 
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3.4.  Feasibility  study  of  IV  image  overlay  and  laser  guidance 
system 

A  volunteer  trial  was  undertaken  to  evaluate  the  feasibility  of  the 
system.  The  feasibility  study  was  approved  by  the  ethics  commit¬ 
tee  the  university.  We  performed  CT  scans  of  a  human  knee.  The 
volumetric  CT  images  for  the  knee  (512  x  512  pixels  x  347  slices, 
thickness  of  0.5  mm)  were  segmented,  and  the  results  were  ren¬ 
dered  and  transmitted  to  the  IV  display.  We  integrated  the  IV 
segmentation  image  in  a  simulated  knee  surgery  and  superimposed 
the  IV  image  on  the  actual  position  (Fig.  11a).  The  position  of  the 
insertion  point  and  the  orientation  of  the  intersection  line  were  cal¬ 
culated,  and  the  laser  planes  were  adjusted  by  rotating  the  motors. 
Fig.  lib  shows  the  procedure  of  alignment  of  the  surgical  instru¬ 
ment,  and  Fig.  11c  shows  the  image-patient  registration  results. 
We  could  align  the  surgical  instrument  with  the  laser  beam  planes 
intersected  laser  line  from  the  side  of  the  device  (Fig.  12a)  or  from 
the  viewing  window  of  the  IV  image  overlay  device  (Fig.  12b).  The 
feasibility  study  showed  that  the  system  provided  both  intuitive 
3D  information  about  the  anatomical  structure  of  the  organ  and 
guidance  for  the  surgical  instrument. 

4.  Discussion  and  conclusion 

We  developed  a  surgical  navigation  system  that  combines  an 
autostereoscopic  image  overlay  technique  with  a  laser  guidance 
technique.  An  actual  3D  image  is  superimposed  onto  the  patient 
by  using  a  semi-transparent  display  based  on  integral  videography. 
The  laser  guidance  technique  is  used  to  show  the  insertion  path  of 
a  linear  surgical  instrument.  Experimental  findings  indicated  the 
feasibility  of  the  system  with  the  laser  guidance  technique  and 
IV  image  overlay  technique.  The  accuracy  of  laser  guidance  was 
2.48  mm  on  average.  Although  the  accuracy  does  not  quite  meet 
the  requirement  for  the  orthopedic  surgery  (less  than  1  mm),  with 
improvement  and  further  evaluation,  the  system  may  become  sat¬ 
isfactory  for  precision-guided  knee  joint  surgery.  The  orientation 
error  was  2.96°,  and  the  standard  deviation  was  2.12°.  The  preci¬ 
sions  of  the  laser  shooter  and  motors  can  be  increased  to  reduce 
these  figures  to  an  acceptable  level  for  orthopedic  surgery.  It  will 
then  be  possible  to  register  an  anatomical  object’s  configuration 
with  an  intra-operative  surgical  instrument  with  high  precision 
through  real-time  IV  image  rendering.  Moreover,  for  soft  tissue  or 
deformable  tissue  around  the  bone,  the  patient-IV  image  registra¬ 
tion  should  be  corrected  by  sensing  intra-operative  tissue  images 
and  automatically  updating  the  displayed  image.  One  of  the  solu¬ 
tions  is  to  fix  optical  markers  to  the  bone  and  calculate  the  thickness 
of  the  soft  tissue  from  the  scanned  images.  The  position  of  the 
crossing  point  and  the  orientation  of  the  guidance  laser  line  can 
be  corrected  to  fit  the  shape  of  the  soft  tissues. 

The  viewing  areas  of  the  image  overlay  and  laser  guidance  were 
limited.  If  we  want  to  change  the  observed  area,  the  device  must 
be  moved  and  rotated.  Fortunately,  the  optical  markers  attached  to 
the  overlay  device  can  be  used  for  tracking  the  position  and  orienta¬ 
tion  of  the  device,  and  the  IV  images  can  be  updated.  Furthermore, 
the  projected  direction  of  the  laser  beam  could  be  automatically 
changed  to  correspond  to  the  position  and  orientation  of  the  overlay 
device. 

The  guidance  area  of  the  laser  should  be  enlarged  and  the  pro¬ 
jected  laser  beam  planes  should  be  mounted  on  a  long  arm  so  that 
the  laser-plane  intersection  line  can  be  controlled  over  a  wider  area. 
More  driven  devices  for  rotation  or  translation  should  be  installed  to 
increase  degrees  of  freedom  of  the  laser  beam  planes.  The  freedom 
of  Moreover,  one  laser  line  can  provide  guidance  for  only  one  sur¬ 
gical  tool.  Guidance  for  multiple  targets  and  surgical  instruments 
might  be  able  to  be  provided  by  using  multiple  laser  shooters  or 
multiple  planes  laser  projectors.  The  thickness  of  the  intersected 


laser  lines  relative  to  their  orientation  to  the  skin  surface  should  be 
considered,  although  the  change  is  small  because  the  guidance  area 
is  small. 

In  conclusion,  our  precision-guided  surgical  navigation  system 
can  superimpose  a  real  3D  image  onto  a  patient’s  body  for  accu¬ 
rate  and  less-invasive  surgery  and  intuitive  guidance  of  a  linear 
surgical  instrument.  Experimental  results  show  that  this  precision- 
guided  system  can  support  surgeons  during  their  operations.  Our 
system  is  simple  and  it  quickly  provide  an  intra-operative  infor¬ 
mation  about  the  anatomical  structure.  It  enables  the  surgeons  to 
identify  the  insertion  path  of  the  surgical  instrument  intuitively. 
We  believe  that  the  accuracy  of  real-time  projected  point  location 
can  be  improved  by  using  a  display  device  with  a  higher  pixel  den¬ 
sity  and  a  laser  guidance  device  with  higher  precision;  hence,  this 
system  should  be  of  practical  use  in  orthopedic  surgery  and  other 
medical  applications. 
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